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Device and method for 3D imaging 



Field of the Inventi on 



5 The present invention is in the field of 3D scanning 

and 3D image reconstruction using structured light. ■ 

Background of the Inventj on 

10 Structured light methods have been used to perform 

depth measurements in microscope images and to obtain 3D 
mask images of the external surfaces of microscopic and 
macroscopic' objects. These methods are broadly based on 
either trigonometric considerations or an understanding of 
15 the frequency transfer characteristics of a lens. The 
latter methods involve the evaluation of patterned images, 
which are obtained by projecting substantially periodic 
structures by means of a grating onto ah object. Commonly 
to achieve this, gratings are imaged in the object planes 
20 and displaced by integral fractions of the grating 
constant. An image is recorded for each position of the 
grating and the images are recombined. We choose to 
describe this physical method Structured Modulation Imaging 
(SMI) . 

25 

WO-A-98/45745 relates to a microscopy imaging apparatus and ' 
associated imaging method and describes the use of 
structured light for generating an optically sectioned 
image of a specimen. In this method mask images are' 
30 generated and only the in focus parts . -of ^tne ., image 
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generated on the object by the grating are used for the 
image reconstruction. 

In DE 19930816 Al a similar SMI method is described in 
5 which surface information is determined by means of a 
projected structure with a periodic brightness, where this 
structure is displaced in steps of 1/n (where n is a whole 
number greater than 2) of the grating constant and the 
projection is captured by means of a 2D CCD camera. An 
10 algorithm is presented that also uses the defocused parts 
of the projected image for the reconstruction of the 3D 
image. 

Both documents have in common that they relate to 
15 microscopy and describe methods in which the surface of an 
object is viewed from a single point of view. Both 
documents require the projection of three or more gratings, 
where each grating is shifted by a known fraction of the 
grating constant. The 3D information of the surface of one 
20 particular side of an object can then be obtained according 
to the methods described in the documents cited above. This 
means these methods relate to the imaging of surface 
structures of a specimen rather than the entire peripheral 
shape of an object. We call these surface structure- images. 
25 obtained from a single point of view x mask' images in that 
they resemble face masks and do not contain 3D information 
from behind the front face of the object (reference is made 
to figure 1 where the difference between peripheral .image 
and mask image is exemplified) . 

30 



There is a need to provide devices that are capable to not 
only reconstruct the 3D mask of a particular plane of an 
object but the three dimensional peripheral shape of the 
object- In addition there is a need to provide devices that 
5 capture the 3D peripheral shape of the object using a 
single projected grid. 

There is also a need to provide devices for peripheral view 
images that are cheap to build and easy and quick to 

10 operate and furthermore easy and convenient to use. There 
is a need for a device that is capable to reconstruct the 
3D peripheral view of an object with one single operational 
step, whereby an operational step is to be understood as 
placing the object in the scanning device, activate the 

15 device and obtain the 3D information of the object without 
any further intermediate steps to be carried out by the 
user. 

The current invention provides an inexpensive, convenient 
20 and safe device and imaging method for reconstructing the 
2L peripheral shape of an object using SMI. It has 
surprisingly been found that a device comprising a 
component suitable to create *mask' type images by 
generating and recording only one "periodic pattern and a 
25 component enabling the creation of masks of at least two 
different sides of an object, enables the construction of a 
3D image of an object in a simple and straightforward way. 
A device according to the invention is cheap to build and 
easy, quick and convenient to operate whereby the 3D 
3" peripheral view of an object can be reconstructed in a 
single operational step. 
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Summary of the invention 

In one aspect the invention relates to a device for 3D 
imaging of the peripheral view of an object using 
5 structured modulation imaging said device comprising: 
-one or more projecting system (SI) generating a 
periodic pattern on the object said system comprising 
one or more light source (L), one or more focussing 
means (Fl) f one or more patterning means (Ml), 
10 - one or more image recording system (S2) comprising a 

focussing means (F2) and one or more means (M2) of 
. recording an image of the periodic pattern generated 
on the object, 

wherein SI, S2 and the object are arranged relative to 
15 each other such that the periodic patterns can be 

generated and recorded for at least two different 
sides of the object, 

In another aspect the invention relates to a method for 3D 
20 imaging of the peripheral view of an object using 
structured modulation imaging comprising 

a) illuminating the object with a light source, 

b) projecting a substantially periodic pattern on the 
. object 

25 c) recording an image of the periodic pattern on the 
object, 

d) optionally recording the wide field image 

wherein steps a-d are carried out for at least two 
different sides of the object: and wherein the data 
30 obtained are analysed to reconstruct the 3D peripheral 

view of the object. 
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In yet another aspect the invention relates to a kit of 
parts comprising the device and one or more software 
package for analysing the recorded information and/or 
5 reconstructing the 3D peripheral view of the object. 

Description of the figures 

Figure 1 shows a representation of a 3D mask image (a) and 
10 a 3D peripheral view (b) . 

Figure 2 shows a schematic of the principle underlying SMI. 
L is a light source, Ml is a grating, Fl is a lens, f is a 
focal position and d is the distance between focal position 
15 and Fl. 

The physical principle that allows the present invention to 
obtain 3D surface distances from structured illumination 
requires a light source (Lj., a patterning means for 
generating a substantially periodic pattern of lighter and 
20 darker bands (Ml) and a focusing means ■ (Fl) . The pattern 
will be seen with maximum contrast between the dark and 
light bands at a given focal position \f) . The contrast of 
the pattern falls off with a known response as a function 
of distance, d, away from f and the banding of the pattern 
25 will eventually become indistinct. The measured modulation 
depth of the pattern on the surface of the object gives a 
direct link to the distance that the surface patch is away 
from the focal position f\ 

3C Figure 3 shows schematic representations of preferred 
embodiments of the projecting system (SI), a detailed 



S1102 1 




description of the figure is given in the section 
describing preferred embodiments of SI. 

Figures 4-6 are schematic representations of preferred 
5 embodiments of the device, A detailed description of the 
figures is provided in the section describing preferred 
embodiments of the device. 

Detailed description of the invention 

10 

The invention relies on structured modulation imaging 
(SMI) . This technique relates the intensity pattern 
(modulation contrast) of a periodic pattern projected on 
the surface of an object to its surface structure. If a 

15 substantially periodic grid of given frequency is projected 
via a lens onto a screen the observed intensity pattern on 
the screen has a modulation contrast that is at a maximum 
at the focal position of the lens. When the screen is moved 
away from the focal plane the observed modulation contrast 

20 on the screen diminishes. Reference is made to Figure 2 
schematically showing the underlying principle. Figure 2 
shows that the modulation contrast of the projected grating 
varies with distance from the focay plane. The relationship 
between modulation contrast and distance is a function of 

25 the lens properties , the spatial frequency of the periodic 
grid and wavelength of the light used. If an object is 
placed within the structured light field, surface features 
of that object that are in the focal plane show a high 
contrast of the projected grating while surface features 

30 away from the plane show a lower modulation contrast. The 
relationship between the distance from focal plane and the 



observed modulation for a lens system is given in 
W098/45745, which is incorporated by reference. In WO-A- 
98/45745 and DE 19930816, which are incorporated by 
reference, it is disclosed that if the grating is displaced 
5 by fractions of the grating constant and an image of its 
projection on the object is recorded for each position of 
the grating, the algorithms such as those* described in WO- 
A-98/45745 and DE 19930816 Al can be used to mathematically 
remove the pattern from the surface and hence reconstruct 
io the 3D image of that surface. Not disclosed previously, we 
have now found that the displacement of the grating is not 
necessary in order to approximate the removal of the 
grating image from the composite image. Here the grating 
image is removed using correlation analysis over one or 
15 more cycles of the grating constant. Typically, the 

correlation analysis returns a depth measurement averaged 
over one or more cycles of the grating. As such the 
correlation analysis provides a good approximation for 
depth when the grating constant is smaller than structural 
20 features, in the object. Optionally, and in addition to the 
composite image, the wide field image (image of the object 
alone) may also be used in the correlation analysis. The 
availability of the wide field image improves the 
approximation of depth measurement, particularly when the 
25 object contains surface detail associated with colour or 
grey scale rather than shape. 

The device and method according to the invention allow the 
rapid 3D imaging of the peripheral shape of an object 
30 within one or more operational step. An operational step is 
herewith defined as placing the object on/in the device, 
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activate the device , obtain the 3D surface information 
(shape) of various different and/or all sides of that 
object. 

In a preferred embodiment of the invention the information 
5 of the 3D peripheral view is obtained in a single 
operational step. This means according to a preferred 
embodiment a user can place the object in the device, 
activates the device and obtains the information of the 
peripheral shape and/or colour of the object without having 
10 to carry out any intermediate steps* 

The functioning of the device according to the invention 
relies on two principles. The first principle (embodied in 
steps a-d of the method according to the invention) leads 

15 to the generation of a mask. The second principle relies on 
masks being generated for at least two different sides of 
the object. More particularly, the second principle relies 
on many narrow masks being generated about an entire 
circumference of the object. The information from the 

20 different masks is combined to reconstruct the peripheral 
view of the object. 

Below, the components of the device according to the 
invention are listed. 

25 

An object according to the invention can be any object of 
suitable size to be placed in the device. Preferably the 
object is a macroscopic object, that means it can be seen 
with the naked-eye. The term ^object" as used herein also 
30 comprises a multiplicity of objects* 
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The device according to the invention comprises one or more 
projecting systems (SI). Each projecting system comprises 
one or more of the following: 

light source (L) , focussing means (Fl), patterning means 
5 (Ml). 

The object is illuminated by one or more light source (L) . 
Any light source can be used, such as for example halogen 
lamps, electric light bulbs, light sources emitting 
10 coherent light, light sources emitting incoherent light, 
lasers or a collimated light source. Preferably the light 
source is a tube lamp. 

A substantially periodic pattern is generated by Ml and 
15 projected on the surface of the object by Fl. 

A substantially periodic pattern is one in which there is a 
clearly dominant frequency component. The frequency 
components of a given spatial pattern can be estimated 
using standard computer means such as application of the 
20 Fast Fourier Transform algorithm. 

The patterning means (Ml) can be for example a grating. Any 
grating known in the art can be used to work the invention. 
The gratings may be linear, planar or spiral, one, two or 
25 three dimensional. An electronic spatial light modulator, 
using transmitted or reflected attenuation schemes may also 
be used. Alternatively, a periodic pattern may also be 
generated by interference of two or more suitably arranged 
conerent light beams, e.g. lasers. 
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Another way of generating a periodic pattern is by 
electronically modulating the intensity of the light source 
L, for example by using a collimated light source, in 
combination with a suitable focusing means (Fl), for 
5 example a suitable mirror and a lens. 

The periodic pattern generated by Ml can be one dimensional 
or two dimensional, stationary as well as movable. 
Preferably, Ml generates a periodic pattern having one 
10 dimensional local periodicity* 

The position of Ml in relation to L, Fl and the object can 
be anywhere as long as the periodic pattern can be 
projected on a surface of the object. 

15 

The focussing means can be any suitable focussing means 
known in the art, such as for examples lenses, mirrors, 
screens or a combination thereof. The lens can for example 
be spherical or cylindrical. In case Fl comprises a mirror, 
20 the mirror can for example be curved or parabolic. 

Fl can be positioned anywhere in relation to L, Ml and the 
object suitably to project the generated periodic pattern 
on the object. 

25 

In a preferred embodiment SI projects a substantially one- 
dimensional (ID) periodic pattern on the object. 



Preferred embodiments of SI dZe shown in figure 3 to which 
3Q reference is made. 
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The preferred embodiment of SI shown in figure 3a comprises 
a light source (L) illuminating a 2D grating (Ml) and 
subsequently the object via a focusing means (Fl). 

5 The preferred embodiment represented in figure 3b shows the 
light source (L) illuminating the object via a 2D grating 
(Ml) and focusing means (Fl), and a screen D having a slit 
to produce a ID patterned line of illumination. The angle 
of the grating with respect to the slit and/or the 

10 direction in which the grating is moved to provide two 
means for altering the spatial frequency of the ID 
illumination pattern, thereby allowing altered modulation 
contrast characteristics for a fixed focusing means system. 
In the case where the grid is parallel to the slit, the 

15 slit should be sufficiently .wide to permit one or more 
cycles of the grid to be projected. 

Figure 3(c) represents another preferred embodiment of SI 
comprising a tubular or essentially linear light source 
20 (L), 

The preferred embodiment of SI represented by figure 3(d) 
comprises a collimated light source (L) imaged via the 
focusing means (Fl) comprising a cylindrical lens to. 
25 produce a narrow line of illumination on the object. Where 
the grating is positioned parallel to the slit the width of 
the projection, should be at least one cycle of the grating 
constant . 

30 The image of the periodic pattern, which is projected on 
the object, is recorded by one or more recording system S2. 
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S2 comprises at least one recording means M2 and at least 
one focussing means F2. 

M2 can be any detector capable of receiving, digitizing and 
5 storing the information obtained in the image of the 
projected pattern, M2 can for example be a 1 or 2 D array 
of opto-electronically active picture elements such as are 
known in the imaging art, e.g. CCD, CMOS or CIS devices. 
Preferably, the image-recording system is suitable to 
10 receive and record also colour information. This allows not 
only to reconstruct the peripheral shape of the object but 
also its colour. Preferably the depth recording and colour 
recording are made separately. Preferably, M2 is a CCD 
device . 

15 The imaging device can be operated as a continuous 
acquisition of the patterns such that each image represents 
an integration of the image over a small time period as the 
pattern is being moved. Alternatively, the images can be 
acquired after each movement. Images may overlap. 

20 

F2 can be any focusing means known in the art as for 
example lenses, mirrors, screens or a combination thereof. 
F2 may also be a mirror, preferably a parabolic mirror. F2 
can for example be a lens of a CCD camera. 

25 

The following sections describe the grating removal 
processes. 

Process 1: In order to mathematically remove the projected 
30 pattern from the surface and to apply correlation analysis 
algorithms without a separate wide field image one cycle of 
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the grating is required. Over one cycle of the grating all 
structural features of the object are assumed constant 
Examples of structural features include, colour or grey 
scale, angle of inclination and depth. In this way one 
5 cycle of the grating yields a minimum, maximum and two mean 
intensities. Correlation analysis using these intensities 
yields an approximate depth measurement averaged over the 
grating cycle. 

10 Process 2: In order to mathematically remove the projected 
pattern from the surface and to apply correlation analysis 
algorithms with a separate wide field image one cycle of 
the grating is required. Over one cycle of the grating some 
structural features are assumed constant, examples include 
15 angle of inclination and depth. In this way one cycle of 
the grating yields a minimum, maximum and two mean' 
intensities. Correlation analysis using these intensities 
yields an approximate depth measurement averaged over the 
grating cycle. 

20 

Process 3: In order to mathematically remove the projected 
pattern from the surface and to apply algorithms such as 
described in WO-A-98/45745 and DE 19930816 Al overlapping 
mask images are recorded in the following way: 

25 A: Mask image 1 is recorded 

B: SI moves relative to the object by a known fraction of 

the grating constant. 

C: Mask image n+1 is recorded 

D: Steps B and C are repeated for n>2. 

30 E: Grid removal algorithms are applied where the wide field 
overlaps in the n~images. 
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Movement of SI relative to the object can be carried out 

t 

either by moving the object or by moving SI. 

It is noted that DE 19930816 Al refers to displacements by 
5 integral fractions of the grating constant. It has been 
found that in the device according to the invention, the 
displacements can be carried out at any fractions of f the 
grating constant. ? 

It is further noted that n according to the above in 
10 principle only need to be equal to two and not greater than 
two. If n=2 the reconstruction of the image is approximate. 
In order to obtain an exact reconstruction we require n>2. 

The device may further comprise a turntable on which the 
15 object is placed. The turntable may be moved 
electronically, mechanically, stepwise or continuously. The 
turntable may be transparent allowing also the side on 
which the object rests to be scanned by the device. 
The turntable may be rotated between 0 and 360 degrees 
20 within one or more planes. 

SI and S2 are arranged such that no triangulation is 

required in the image analysis process. This may be 

achieved by collocating SI' and S2. This may also be 
25 achieved with the aid of mirrors. 

In order to obtain the peripheral view of an object, masks 
from at least two different sides of the object need.. to be 
constructed. In order to achieve this in a rapid and 
30 convenient way SI and S2 are arranged relative to the 
object so that masks can be constructed of at least two 
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different sides of the object. Preferably many narrow masks 
are measured, this can be achieved in a number of ways. 

The device may for example comprise a turntable on which 
5 the object is placed. The turntable may be rotated stepwise 
or continuously, mechanically or electronically in relation 
to SI and S2. in this embodiment the device preferably 
comprises only one SI and one S2, which are at fixed 
positions. By moving the turntable between 0 and 360 
10 degrees (not including 360 degrees) the object can be 
imaged peripherally. 

Alternatively, SI and S2 may be in a fixed position 
relative to each other but capable to be moved around 
15 and/or over an object either continuously or stepwise, 
mechanically or electronically, in this embodiment, the 
object is preferably at a fixed position, although it is 
also possible to include a turntable in this device. 

20 in another embodiment according to the invention the device 
comprises a plurality of projecting systems and recording 
systems at various positions, each combination of SI and S2 
facing different sides of the object. The systems may be at 
fixed positions or movable. 
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According to another embodiment, the device is configured 
as to comprise a combination of the above. 

in oraer to image also the side on which the object rests 
10 optionally a transparent turntable is used or a second 
operational step is introduced wherein the object is turned 
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such that it rests on a different side and the scanning 
process is repeated. 

In order to reconstruct the 3D information of one view of 
5 the object (to generate a mask) algorithms such as 
described in DE 19930816 Al or correlation analysis is 
applied for which the device preferably comprises suitable 
software. Preferably, the device is also provided with 
suitable software to recombine the masks for reconstructing 

10 the peripheral shape of the object. Furthermore, the device 
is optionally provided with means to display the generated 
peripheral view. The device may also be used as computer 
peripheral and is then provided with a means to record and 
store the information such that the information and/or the 

1£ peripheral view can be processed, generated and displayed 
on a separate computer. The invention therefore also 
relates to a kit of parts comprising the device and a 
software package for analysing processing and displaying 
the surface information. 

20 

In another aspect the invention provides a method of 3D 
imaging of the peripheral view of an object. Parts of the 
method have already been illustrated during the description 
of the device. The method according to the invention 
25 comprises 

a) illuminating the object with a light source, 

b) projecting a substantially periodic pattern on the 
object 

c) recording an image of the periodic pattern on the 
30 object, 

d) optionally recording the wide field image 



wherein steps a-d are carried out for at least two 
different sides of the object and wherein the data obtained 
are analysed to reconstruct the 3D peripheral view of the 
object. 

5 

This is further demonstrated by preferred embodiments, 
which are not to be understood as limiting. ■ • 
In one embodiment steps a-d are carried out for one 
particular view of the object. Steps a-d are then repeated 
10 for other views of the object and the combined data is 
analysed for the reconstruction of the peripheral view. . 

In another embodiment- a plurality of systems SI and S2 
record images of periodic patterns that have been generated 
15 at different sides of the object. After these images have 
been recorded the systems are moved relative to the object 
and the wide field images are recorded. 

Prior to use the device can be calibrated. Calibration can 
20 be carried out once and the device is set up or calibration 
can be carried prior to every single use. 

Calibration of the device can for example be provided as 
follows . 

A calibration means of known dimension is inserted into the 
25 device and the 3D scanning process is carried out. The 
observed values of modulation ratio for each surface patch 
are then cross-correlated with the known distances that 
exist for each surface patch. The results of the 
correlation are stored in a look up table so that during 
30 normal operation an observed modulation ratio value is 
transformed into distance before further data processing. 
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Alternatively all data processing can be carried out in 
modulation ratio values and after processing transformed 
into real distances. 

5 In a preferred embodiment of the invention a calibration 
figure is inserted on, above or below a turntable and fixed 
so that on rotation of? the turntable the figure rotates. 
The observed values of modulation ratio for each surface 
patch are then cross-correlated with the known distances 

10 that exist for each surface patch on the calibration 
figure. The results of the correlation are stored in a look 
up table so that after a complete revolution of the 
turntable the observed modulation ratio values can be 
transformed into distances before further data processing. 

15 By this arrangement each full 3D scan is self calibrated 
without any further calibration steps. 

The invention is now further described by a number of 
preferred embodiments. 

20 

Embodiments of the Device 
Embodiment I 

A schematic of a. preferred embodiment of the device is 

25 shown in figure 4. 

In this embodiment of the invention SI is described in 
figure 3d, i.e. comprising a tubular lamp as a light source 
(L), a parallel grating (Ml), a screen with a slit parallel 
to the long axis of rne light source (D) and a cylindrical 

30 lens (Fl) . The device comprises a turntable on which the 
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object can be placed (indicated by bent arrow) . S2 is a CCD 
camera . 

A preferred way of operating this device is as follows: 
The object is illuminated with one or more cycles of the 
5 periodic pattern, and recorded. The object is then rotated 
by a fraction of one cycle, one cycle, or more than one 
cycle of the periodiq pattern. This process is repeated 
until the object has been circumnavigated once. The data of 
the different masks are combined for reconstruction of the 
10 peripheral view of the object. Optionally the wide field 
. image without grating is recorded by the same process. 

Embodiment II 

15 This embodiment is similar to embodiment I. m this 
embodiment S2, a CCD camera, is attached to SI which is the 
preferred embodiment shown in figure 3d, and both SI and S2 
are moved around and/or over the object thus obtaining 
surface information of several sides of the object. This 
20 embodiment is represented in figure 5. 

Embodiment III 

This embodiment comprises three projecting and recording 
25 systems (SI and S2) arranged at different positions in the 
device so that each combination of projecting and recording 
system faces a different side of the object. 
A representation of this embodiment is shown in figure 6. 



S1102* 



20 



Claims 

1. A device for 3D imaging of the peripheral view of an 
object using structured modulation imaging comprising: 
-one or more projecting system (SI) generating a 
periodic pattern on the object , said system comprising 
one or more light source (L), one or more focussing 
means (Fl) f one or more patterning means (Ml) , 

- one or more image recording system <S2) comprising a 
focussing means (F2) and one or more means (M2) of 
recording an image of the periodic pattern generated 
on the object, 

wherein SI, S2 and the object are arranged relative to 
each other such that the periodic patterns can be 
generated and recorded for at least two different 
sides of the object, 

2. A device according to claim 1 wherein the object is a 
macroscopic object, 

3. A device according to any preceding claim comprising 
software for analysing the recorded information and/or 
reconstructing the 3D image of the object. 

4. A device according to any preceding claim wherein M2 
is a CCD camera. 

* 

5. A device according to claim 6 wherein F2 is a lens of 
the CCD camera 
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A device according to any preceding claim further 
comprising a turntable on which the object can be 
placed. 

A device according to any preceding claim comprising a 
Plurality of SI and S2 in a fixed spatial arrangement 
each combination of SI and S2 facing different sides 
of the object. 

A device according to any of claim 1-6 comprising one 

51 and one S2 in a fixed spatial position and a 
turntable which can be turned in relation to SI and 
S2. 

A device according to any of claim 1-6 wherein SI and 

52 are fixed in relation to each other and wherein SI 
and S2 can be moved around and/or over the object. 

A device according to any preceding claim wherein the 
periodic pattern generated has local one dimensional 
periodicity. 

A method for 3D imaging of the peripheral view of an 
object using structured modulation imaging comprising 

a) illuminating the object with a light source, 

b) Projecting a substantially periodic pattern, on 
the object 

c) recording an image of the periodic pattern on the 
object, 

d) optionally recording the wide field image without 
the periodic pattern 
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characterised in that steps a-d are carried out for at 
least two different sides of the object and wherein 
the data obtained are analysed to reconstruct the 3D 
peripheral view of the object. 

12. A method according to claim 11. wherein the object is a 
, macroscopic object. 

13. A kit of parts comprising a device according to claim 
1 and one or more software package for analysing the 
recorded information and/or reconstructing the 3D 
peripheral view of the object. 
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Abstract 



A method and device are described for 3D imaging of objects 
whereby an object is illuminated by a light source and a 
periodic pattern is generated on the object. The image is 
recorded and analysed to remove the spatial pattern from 
the images-. This is carried" 1 out: for different sides of the 
object to obtain the full 3D image of the object! 
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